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Abstract 

An Ashkin- Teller neural network, allowing for two types of neurons is con- 
sidered in the case of low loading as a function of the strength of the 
respective couplings between these neurons. The storage and retrieval of 
embedded patterns built from the two types of neurons, with different 
degrees of (in) dependence is studied. In particular, thermodynamic prop- 
erties including the existence and stability of Mattis states are discussed. 
Furthermore, the dynamic behaviour is examined by deriving flow equa- 
tions for the macroscopic overlap. It is found that for linked patterns 
the model shows better retrieval properties than a corresponding Hopfield 
model. 
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1 Introduction 

One of the best known physical models for neural networks is the Hopfield 
model 0. In theoretical investigations of network properties, e.g., the retrieval 
of learned patterns, it plays a similar role as the Ising model does in the theory of 
magnetism. Extensions of this model to multi-state neurons have received a lot 
of attention recently (see, e.g., [@] - and the references cited therein). Thereby 
the ability to store and retrieve so-called grey-toned and coloured patterns has 
been investigated. 

In this work we consider another extension of the Hopfield model to allow 
for multi-functional neurons. The specific model we have in mind is the neural 
network version of the Ashkin- Teller spin-glass (0-0). Indeed, on the one hand 
the Ashkin- Teller model has two different kinds of neurons (spins) at each site 
interacting with each other. This allows us to interprete this model as a neural 
network with two types of neurons having different functions. On the other 
hand, this Ashkin- Teller neural network (ATNN) can be considered as a model 
consisting out of two interacting Hopfield models. 

We expect the behaviour of the ATNN to be different from the one of the 
Hopfield model in a non trivial way. One of the things we want to find out, e.g., 
is whether this (four-neuron) interaction between the two types of neurons can 
improve the retrieval process for embedded patterns built from these two types of 
neurons. We will see, indeed, that for a particular choice of this interaction term 
the retrieval quality of the embedded patterns is very high in comparison with a 
corresponding Hopfield model. Therefore, independent of the possible biological 
relevance of this model, if any, such a study is interesting from the pure physical 
point of view. 

In this work we consider both the thermodynamic and dynamic properties of 
this model in the case of loading of a finite number of patterns. 

The rest of this paper is organized as follows. In section [| the ATNN model 
is introduced. Section ^ discusses the methods used for analyzing both the equi- 
librium properties and the dynamics of the model. In particular, fixed-point 
equations as well as flow equations for the relevant macroscopic overlap order 
parameters are derived. In section |] numerical solutions of these equations are 
discussed for a representative set of network parameters. The retrieval proper- 
ties of embedded patterns with different degrees of dependencies are compared. 
Section [| presents the main conclusions. 
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2 The model 

We consider a network of N sites. At each site we have two different types of 
binary neurons, and cr^ , i — 1, . . . , N. The two types of neurons interact via a 
four- neuron term SiSj<Ji<jj. The infinite-range hamiltonian reads 

H = -- X)[./£ )s < s J + 4 2,<7 *°J' + Ji? SiSjViaj] (1) 

In this network we want to store a finite number of patterns, p, also of two 
different types, i.e., = [A = 1, . . . ,p and 77, = {77^ }, z/ = 1, . . . ,p, which are 
supposed to be independent identically distributed random variables (i.i.d.r.v.) 
taking the values +1 or —1 with probability 1/2. To build in the capacity for 
learning and retrieval in this network its stable configurations must be corre- 
lated with the configurations determined by the learning process. This can be 
accomplished by taking the Hebb learning rule for the interactions 

4 } = ^E^, 4? = ,UE«, J if = ¥ J 3E 7f 7i , (2) 

/^=l ^=1 n=\ 

where the 7^ = {j? }, /x = 1, . . . ,p are also i.i.d.r.v. taking the values +1 or — 1 
with probability 1/2. 

At this point some remarks are in order. Firstly, we have taken the strength of 
the two types of patterns to be equal, meaning that the ATNN model is isotropic. 
Secondly, it is clear that the behaviour of this model (HD-(PD might depend on the 
fact whether the 7 are taken to be independent from the £ and the 77 or not. The 
following cases will be distinguished: 

1. unlinked patterns 

(a) £j,T7j and 7 i are i.i.d.r.v. 

(b) = 7?j and 7^ are i.i.d.r.v. 

(c) & = m = 7; is i.i.d.r.v. 

2. linked patterns 

li = ZiVi wi th £i and r) { i.i.d.r.v. 
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We note that this ATNN model can also be considered as an assembly of two 
single Hopfield models (when J3 = 0), one in the Sj-neurons and one in the <7j- 
neurons interconnected via a four- neuron interaction (when J 3 7^ 0). The study 
of coupled Hopfield networks has aroused some interest in the literature before 
(e.g., 0). 

In the following we discuss both the thermodynamics and the dynamics of 
this ATNN neural network with low loading. 



3 The method 



3.1 Statics 

Starting from the hamiltonian (0)-© and applying standard techniques (lin- 
earization and the saddle-point method ||TT|-|T^[) the ensemble-averaged free en- 
ergy is given by 

/ = -(Jim? + J X Ya\ + J 3 m') 

— — ((ln[4 cosh (3L\ cosh (3L 2 cosh (3L 3 (l + tanh (5L\ tanh /3L 2 tanh /3L 3 )])) 

(3) 

with 



L 1 = J 1 £-m 1 , L 2 = Jirj-m 2 , L 3 = J37 ■ m 3 . (4) 

In the above the double brackets ((■)) denote the average over the distribution of 
the embedded patterns. The m Q = {m 1 ^}, \i = 1, . . . ,p; a = 1, 2, 3 are, as usual, 
overlap order parameters defined by 

^ N 1 N -t N 

i=l i=l i=l 

Here we remark that in the thermodynamic limit N —>■ 00 and for finite loading 
a = the diagonal terms in the couplings, Ju, do not play any role in the 
Hamiltonian ([[]). 

In fact our model can be considered as a special case of the general spin-glass 
model presented in ref. @] such that the expressions can also be read off 

from there. 
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The fixed-point equations for the order parameters read 

_ // i/> a (tanh (3L a + tanh /?L„ tanh j3L p ) \\ 
ma ~~ \\ 1 + tanh /3L Q tanh /3L U tanh /5L p / / ' ^' 

where a,u,p = 1,2,3 are taken to be different and where tp a is the embedded 
pattern corresponding to m a . 

Since the study of this ATNN model is very involved we have restricted our- 
selves here to a detailed treatment of the Mattis states (|I2|) which are especially 



important from a neural network point of view. In our case they are defined as 
those solutions of the fixed-point equations for which not more than one com- 
ponent of each order parameter is different from zero, e.g. rri! = mi(l,0, ...,0), 
m 2 = m 2 (l, 0, 0), m 3 = m 3 (l, 0, 0). These states are denoted by miro 2 m3 
in the sequel. We will see that those states are the only ones which contribute 
to the thermodynamics of the system. Solutions with more than one component 
being non-zero, i.e., mixture states will be important for the dynamics if they are 
local minima of the free energy. 

For zero temperature we note that the eqs. (|6p can be simplified by replacing 
the tanh (3L a by signL a . Then it is straightforward to show that for each order 
parameter 

(m Q ) 2 <l, a = 1,2, 3, (7) 

with the equality being satisfied for a one-component m a , and that the ground- 
state energy is given by 

E=-i:\™i. (8) 

In section |] we report on the existence and stability of these Mattis states as a 
function of the temperature. 



3.2 Dynamics 

The analysis outlined above enables us to find the local minima of the free 
energy. But in order to find out how an arbitrary initial state of the network 
changes in time and to what extent, if at all, one of the learned patterns is 
approached, we derive a flow equation for the overlap order parameters. 

We consider sequential updating of the spins consistent with the detailed 
balance condition. Hence we choose the following transition probabilities for a 
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spin flip at a certain time step 

ui(si, <7i) = uj(si -> -Si) = — [1 - tanh(/5(s i /i- 1) + s^/if ^)] 

w 2 (si, (Ti) = u)((Ti -> -a*) = —[1 - tanh(/5(ai^ + s^/i} ; ))] 

w 3 (s is o-j) = w(s< -> -Si, o"i -> -a*) = [1 - tanh(^(s i /if } + o^/if ^))] (9) 

with the /ij appropriate local fields acting on the neurons in the following way 

W 4eEW- ^ (2) =|eE*. ^ (3) = #EE7fT^, (io) 

3 ft j fl JV j ft 

where, as in the treatment of the statics we take J\ = J 2 . At this point we remark 
that in the thermodynamic limit A" — > oo the diagonal terms in the couplings, 
Ja, do not survive. Furthermore we note that the method used here is also valid 
in the case of unequal J. 

We then consider the probability p(s, cr; t) that the system is in a state s = 
(sj, s^), cr = (o"i, ctat) at time t. It satisfies the master equation 
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- p(s,a;t)[uj 1 (s i ,a i )+U2{s i ,a i )+uj z (s i ,a i )]^ . (11) 

The operator v = 1,2,3 acting on a configuration {s, cr} changes the sign of 
the following spins: Sj for v — 1, &i for v = 2 and simultaneously s« and <Ji for 
v = 3. 

/,From this we want to derive a flow equation for the overlap order parameters. 
Because of the multi-state character of the model (due to the four-spin interaction 
term) the summation over i has to be carried out by generalizing the method of 
submagnetizations or suboverlaps connected with partitions of the network with 
respect to the built-in patterns |jTB[-|T^|. We note that for Hopfield networks we 



do not need such a partitioning [15 



First we introduce the following division of the network indices 

{t<N} = [jl k I k = {t<N;k = k l } k i = (£ i ,n i ,~f i ). (12) 

k 
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Then we define the so called submagnetisations or suboverlaps 

A*Q!,k(s, &) = — — r ^ Sa,i « = 1,2, 3 Si ti = Si, S 2 ,i = (Ti, S 3j i = Si(Ti, (13) 

which enables us to write the overlaps in the form 



ni 



a = T.^^k {a - l)p+u a = 1,2,3, u = l,...,p, (14) 



where stands for the number of indices in the set Ik- The number of vectors 
k is equal to 2 ap , which is much smaller than the number N of sites % (when 
N — > oo). The coefficient a = 3, 2, 1 and 2 for patterns of type 1(a), 1(6), 1(c) and 
2 respectively. So, an embedded random pattern configuration can be assumed 
to be equally distributed over the sets Ik such that \Ik\ = 2~ ap N + 0(N 1 / 2 ). 

Next we write down the probability P(/x; t) that the system is in a macroscopic 
state described by a set of submagnetisations /x(s, <x) = {/i Q ,k} 

P(m; t) = p( s ' - Ms, °0) • ( 15 ) 

{s,<7} 

Then we arrive at the following master equation (cfr. eq. (|TTD) 
ot {s,cr} i v 

(16) 

The action of the operator F? can be specified further by writing 

A*(i?{s, <t}) = /x(s, a) - R"( Si , a,) . (17) 

It is then straightforward to check that certain elements of the matrix [R I/ (sj, <7j)] aj k 
are zero, viz. 

4 = 4k = 4k = () vk,*<iv. (is) 

Furthermore, for v and a different from these specific values in ([TJD we have 

K k = TTT S <x,i ^ i elk (19) 

|-*k| 

i£ k = ifz^/k. (20) 
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In the thermodynamic limit iV — > oo the parameters \x become continuous vari- 
ables. Following [jn| we then write for an arbitrary smooth function <f)(fi) 



dim 

dt 



dP(nt) 
dt 



{ S ,cr} 



x [0O' - R u (s i} <Ti)) - 4>{V 



(21) 



Making an expansion of <p around [i and doing a partial integration with respect 
to \x we arrive at 



9jm 

dt 



J a,k a ^a,k \{s,<X} i f 

xi^ k (s 4 , a,) ^(si, ^ + OiN' 1 ) . (22) 

We remark that up to now we have not used the specific form for the transition 
probabilities u u . Other expressions for u u , satisfying detailed balance could also 
be employed. 

Using the partitioning of the network into subsets Jk we can replace the sum 
over i by Z)kHie/ k - Employing the specific form (^) for to v and performing the 
sum over {s, cr} we arrive at (compare |13| ) 



dt 



«,k °vvk 



;^Q,k - /a,k({^/3,k}) 



(23) 



where / a ,k({/^/3,k}) is a function of all the suboverlaps given by 

= ^{tanh(/5L Q + (3L U ) + tanh(/3I a - (3L V ) + tanh(/?I a + (3L p ) 

+ tanh(/3L Q , — f3L p ) + /ij, i k[tanh(/3L Q + (3L P ) — tanh(/3L Q — f3L p )] 
+// P)k [tanh(/3I Q + (5L V ) - tanh(/3I Q - /Sir,,)]} (24) 



/a,k({^,k}) 



with the L a (a, u, p = 1, 2, 3 and different from each other) given by eq. (f|). Since 
this equation holds for every smooth function the equation for the probabilities 
P(fi;t) has the form 



9* 



a,k 9 ^a.k 



Ma,k - /a,k({^,k}) 



(25) 
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and the corresponding flow equations for the submagnetisations n a ,k themselves 
read 

= ~3^ a ' k + /«.k({/i/3,k}) • (26) 

Together with ( fL4|) these coupled equations are used to study the dynamic 
behaviour of the network. In the following section the results of this study are 
discussed. 



4 Results 

In this section we discuss the numerical results for the ATNN model obtained 
from the fixed-point equations specifying the thermodynamic properties and from 
the flow equations for the suboverlaps describing the dynamics. We treat the cases 
of linked patterns and unlinked patterns separately. We report the results for a 
set of representative examples illustrating the main new features of the model. 



4.1 Unlinked patterns 

We first consider the model with equal coupling parameters J3 = J±(— J 2) = 
1. Introducing Mattis-type states mim 2 m 3 into the fixed-point equations @ 
we obtain the same equations for m Q , a = 1,2,3 irrespective of the degree of 
dependence between the different types of patterns. The solutions are presented 
in Fig. |l| where the overlap for different Mattis states is shown as a function of the 
temperature T = 1/(3. A stability analysis performed by studying the stability 
matrix given by = dr ^Q m u leads to the following main features. Above 

T = 1.0 there exist no stable Mattis states. Furthermore, the (paramagnetic) 
state 000 where all the overlaps with the embedded patterns are zero is stable. 

Below T = 1.0 we have the retrieval phase with many different forms of stable 
Mattis states. We expect that the most important ones are those with the lowest 
energies, i.e. the states mmm (for temperatures in the interval (0.83, 1)) and 
mmO (for temperatures in (0,0.83)). The state mmO corresponds to a situation 
where the overlaps with a pattern in the s-part and a-part of the network are 
non-zero and equal to each other and the overlap with a pattern in the sa-part 
of the network is zero. This state is completely equivalent to the states mOm and 
0mm, a fact resulting directly from the symmetry of the model. Its properties 
are analogous to those of the Mattis states in the Hopfield model ||12||, since the 



fixed-point equations for the overlap m are the same. The states which have 
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no analogues in the Hopfield model are mmm and also rami. The first can be 
interpreted as the description of retrieval of one pattern, "simultaneously by the 
s, o and sa-parts of the network" (for the case of dependent patterns, i.e., case 1 
(c), this pattern is the same for the three parts). However, such a retrieval occurs 
with a lot of errors as can be inferred from the small values of the corresponding 
overlap in Fig. [l|. The second state, i.e. mml, which is, of course, equivalent 
to mlm and Imm differs from the mmm state in this respect that the non-zero 
overlaps with the pattern in the three different parts of the network are not equal 
to each other. But this state does not seem to play an important role because it 
is never a global minimum in the set of Mattis states (see Fig. [I]). 

Increasing the temperature to T = 1 we notice a continuous transition from 
the network retrieval phase to the disordered (paramagnetic) phase. 

Next, we have also studied the local minima structure for non-equal values of 
the coupling parameters Ji, J 2 and J 3 . It only differs in a quantitative way. 

The static results found above are confirmed by a study of the dynamic be- 
haviour of the ATNN model using the coupled eqs. (|26f) . For simplicity, we take 
the number of embedded patterns of each type p = 2. Since we have three dif- 
ferent kinds of such patterns the results concern a six-dimensional flow. Some 
representative two-dimensional projections are presented in Fig. |2|. 

As a starting point we take ni2 = 1x13 = (0.5,0) and different values for mi. 
Such a choice of initial conditions is not very specific because of the symmetry 
properties of the model. It allows us to show some typical behaviour of the 
network. An extensive search confirms that other initial conditions lead only to 
quantitatively different diagrams. We remark that the part of the diagrams not 
shown explicitly is symmetric with respect to the m\ or m\ axis. We choose 
some relevant values of T suggested by the thermodynamics. We can locate 
in the first diagram of Fig. |2| (T = 0.1) the attractor 0mm in the lower left 
corner. In the lower right (and, since there is symmetry with respect to the 
off-diagonal also upper left) corner we see the state /mm, which looks like an 
attractor. However, our static analysis reveals that they are only saddle points 
in the full six-dimensional space. On the off-diagonal we have a state of the 
form rri! = mi(l,l), m 2 = m 2 (l,0) and m 3 = m 3 (l,0) denoted by smm, i.e., 
symmetric with respect to the s-part of the network. We remark that in this 
diagram some lines cross each other which is caused by the fact that only the 
evolution of two order parameters is shown whereas the third order parameter, 
m\ = ml, is also evolving. One could easily imagine a three-dimensional picture 
with ml = m\ taken as the third coordinate. 

For increasing T the smm state is no longer present and the Imm states move 
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along the m\-axis (respectively m^-axis) until they disappear for T « 0.8. For 
these temperatures there is a very small difference in free energy between the 
various Mattis-type states. This could be the reason that for T = 0.8 we were 
no longer able to detect the states Imm which should still exist according to the 
static analysis. Indeed, at this temperature the overlap for the Imm states is 
almost equal to the one for the mmm state (see Fig. |lj) showing that they are 
almost identical. Furthermore, for T « 0.8 the states mmm appear, move towards 
the origin on the mj-axis (respectively the m^-axis) as seen on the diagram for 
T = 0.9 and disappear for T = 1.0. From this temperature onwards only the 
origin, i.e. the 000 state is an attractor. 



4.2 Linked patterns 

Next, we have analysed the model with linked patterns satisfying 7^ = £^77^ 
with £ and r) i.i.d.r.v., and with equal coupling parameters J 3 = J\{= J2) — 1- 

Introducing again Mattis-type states m^m^m?, into the fixed-point equations 
(Q) and checking their stability we find that there are only two stable solutions: 
the retrieval state mmm which is stable below T = 1.213 and the paramagnetic 
000 state which is stable above T = 1.0. The corresponding retrieval overlap is 
shown in Fig. [l] (filled symbols). We notice that in contrast to the model with 
unlinked patterns a state of the form mmm has a much bigger overlap. 

Hence, we can distinguish different phases: a retrieval phase below T = 1.213 
and a paramagnetic phase above T = 1.213. We remark that the transition at 
T = 1.213 is first order. In the temperature region 1 < T < 1.213 both the 
paramagnetic and Mattis solutions are local minima of the free energy. Such a 



region has also been seen in the Potts model but not in the Hopfield model [fL3 
Finally, we find that Hopfield-type solutions, i.e., Mattis states of the form mOO 
are (only) saddle points below T = 1.0. 

A detailed study of the flow equations (|26|) reveals a much more complicated 
local minima structure for this model. It turns out that for the model with 
linked patterns we still have to distinguish between the Mattis solutions according 
to the relative place of the non-zero overlap components for the different order 
parameters. Consequently, we introduce simple Mattis states where only the 
same components of the different order parameters are non-zero, e.g., rri! = 
rai(l,0), m 2 = m 2 (l,0), and m 3 = m 3 (l,0) for p = 2, denoted as before by 
777.1777,2771.3. For equal components m a these are the states we have encountered in 
the thermodynamic analysis. (They are equivalent to (0, m)(0, m)(0, m) and also 
to (-m,0)(-m, 0)(m, 0)). 
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Besides, we define crossed Mattis states where never the same components of 
the different order parameters are non-zero, e.g., (m, 0)(0, m)(0, 0) for p = 2 and 
(m, 0, 0)(0, m, 0)(0, 0, m) for p = 3. At this point we remark that a state of the 
form (m, 0)(0, m)(m, 0) is neither simple nor crossed but of a mixed form. For 
p = 2 we did not detect the latter. The crossed states are in fact equivalent to 
Mattis states for a model with unlinked patterns. This can easily be checked 
by introducing this type of solutions in the fixed-point equations for the order 
parameters and taking appropriate averages over the linked patterns. 

Some representative flow diagrams are shown in Fig. |3| and Fig. [|. As before 
we present only projections onto a two-dimensional space. The symmetry of the 
model with linked patterns is different from the model with unlinked patterns 
what results in a different symmetry of the flow diagrams. So, the remaining 
part of a diagram in these figures can be obtained by a reflection of the part 
displayed with respect to the axis m\ = 0. 

The initial conditions for the flow diagrams of Fig. |3] are as follows: two 
identical Mattis states with m2 = 1113 = (0.5,0). We can locate in the first 
diagram of Fig. ^ (T = 0.1) the attractor mmm in the lower right corner. In 
the lower left corner we see the state mOO, which again looks like an attractor. 
However, it is only a saddle point in the full six- dimensional space. On the top 
in the middle we have a state of the form mi = (m\,ml), m 2 = m 2 (l,0) and 
m 3 = ^3(1, 0) denoted by amm, i.e., asymmetric with respect to the s-part of 
the network. We remark that also here any crossings of paths are caused by the 
fact that the diagrams of Fig. |^ and also of Fig. ^ are projections of a higher 
dimensional flow. They are not present in the full six-dimensional space. 

For higher T the state amm disappears, the state mmm stays in the lower right 
corner up to T = 1 and the state mOO moves towards the origin and disappears 
at T = 1. Above T = 1 (see the diagram for T = 1.2) both the origin and the 
state mmm are stable, but the latter has already moved towards the origin. We 
note that in contrast with the model with unlinked patterns, it does not reach 
the origin since the transition (to the paramagnetic phase) is first order. 

Another illustrative set of diagrams is presented in Fig. [|. Here the initial 
conditions are more general: rri2 = (0.1,0), 013 = (0.5,0.0). In the first diagram 
for T = 0.1 an attractor mmm is present in the lower left and right corner. On the 
top in the middle the state mmO is located. It is a crossed state and a minimum 
at low temperatures. The overlap m in this state depends on T in the same way 
as the overlap of a Mattis solution of the standard Hopfield model does. 

For higher T this crossed state disappears but the mmm states nearly stay at 
the same place until T = 1. Above, the origin is an attractor and the mmm states 
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start to move towards the origin (see the diagram for T = 1.2). As explained 
above they do not reach the origin. 

The interesting conclusion of these figures is that the state mmm has a big 
basin of attraction. The latter is, of course, somewhat reduced in the temperature 
region where also the stable state 000 appears. Furthermore, the state mmm has 
a large overlap (recall Fig. [I]) with the embedded patterns. 

Since the ATNN model with linked patterns seems to have very good retrieval 
properties it is worthwhile to derive a (3J% — (3 J3 phase diagram. We first note that 
the fixed-point equations @ for the simple Mattis states have the same form as 
those for the mean-field Ashkin- Teller model. Because these states are always the 
global minima of the free energy they determine the transition lines. Of course 
the meaning of the phases is different from the standard Ashkin- Teller model. 
For the special case of one embedded pattern, i.e., p — 1, with = rjj — 1 both 
models are completely equivalent. 

The ATNN f3J\ — (3J% phase diagram (for low loading of linked patterns) is 
presented in Fig. |5|. We distinguish the following phases. The area in dark grey is 
the paramagnetic 000 phase. The light shadowed area is the full retrieval phase 
(for linked patterns) described by the simple Mattis state mml (m 7^ / when 
J\ 7^ ^3)- The white area represents a partial retrieval phase, i.e., only patterns 
embedded in one part of the network are recalled by the state 00/. The (thick) 
full lines indicate second order transitions, the (thick) dashed lines discontinuous 
ones. We remark that the 000 state exists as a minimum up to the thin full lines, 
but outside the dark grey region its energy is higher than the energy of the mml 
state. In part of the full retrieval phase, namely in the upper half of the phase 
diagram, the crossed state mmO exists. It is stable in the area above the thin full 
and thin dashed lines. 

5 Conclusions 

We have analysed a neural network version of the Ashkin- Teller spin-glass 
model for low loading of patterns. Both the thermodynamic and dynamic prop- 
erties have been considered, especially for Mattis states which are the most inter- 
esting states from the point of view of retrieval. Fixed-point equations as well as 
flow equations for the relevant order parameters have been derived. Numerical 
results have been discussed illustrating the typical behaviour of the network. 

The following main conclusions can be drawn. For unlinked embedded pat- 
terns the behaviour of the model is much richer than in the case of the standard 
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Hopfield model in the sense that many different forms of stable Mattis states 
are possible. These states exist up to T = 1 where a continuous transition oc- 
curs from the retrieval phase to the paramagnetic phase. The corresponding flow 
diagrams are quite complicated but verify the existence of these many attrac- 
tors. However, none of these retrieval states has a bigger overlap than the Mattis 
states of a corresponding Hopfield model. Hence the inclusion of the four-neuron 
interaction term does not particularly improve the quality of retrieval. 

For linked embedded patters interesting new features show up. The most 
important one is that stable Mattis states of the form mmm appear. They have 
a very big overlap with the embedded patterns, meaning that the pairs of patterns 
which are linked by the four-neuron term are retrieved with a very high accuracy. 
Furthermore, they exist up to T = 1.213 and have a big basin of attraction. 
For temperatures 1 < T < 1.213 both these Mattis states and the paramagnetic 
solution are local minima of the free energy such that their basin of attraction is 
somewhat reduced. To verify then that this big overlap m is not just a rescaled 
overlap of the corresponding Mattis state of a corresponding Hopfield model we 
have made a comparison in Fig. |6|. We clearly see the difference in shape in 
favour of the ATNN. Further details of additional features of the ATNN model 
are given in a (3 J\ — (3 J3 phase diagram (see Fig. |5]) . In brief, the linked pairs are 
retrieved easily and with a high precision (simple states) and the unlinked pairs 
may be retrieved (crossed states), but always with lower precision than that of 
the linked ones. It is important to stress that the patterns which are linked can 
be completely different. In principle, they are independent. 

Acknowledgements 

This work has been supported in part by the Research Fund of the K.U.Leuven 
(Grant OT/94/9). The authors are indebted to Marc Van Hulle of the Neuro- 
physiology Department of the K.U.Leuven for interesting discussions concerning 
the possible biological relevance of this model. One of us (P.K) would like to 
thank Prof. G. Kamieniarz for encouragement to study neural networks. Both 
authors acknowledge the Fund for Scientific Research-Flanders (Belgium) for fi- 
nancial support. 



References 



14 



[1] Hopfield J J 1982 Proc. Nat. Acad. Sci. USA 79 2554 

[2] Gerl F and Krey U 1994 J. Phys. A: Math. Gen. 27 7353 

[3] Bolle D, van Hemmen J L and Huyghebaert J 1996 Phys. Rev. E 53 1276 

[4] Kiihn R and B6s S 1993 J. Phys. A: Math. Gen. 26 831 

[5] Bolle D, Rieger H and Shim G M 1994 J. Phys. A: Math. Gen. 27 3441 

[6] Christiano P L and Goulart Rosa Jr S 1986 Phys. Rev. A 34 730 

[7] Moreira J V and Christiano P L 1992 Phys. Lett. A 162 149 

[8] Moreira J V and Christiano P L 1992 J. Phys. A 25 L739 

[9] Nobre F D and Sherrington D 1993 J. Phys A 26 4539 

[10] Viana L and Martinez C 1995 J. Phys. I France 5 573 

[11] Provost J P and Vallee G 1982 Phys. Rev. Lett. 49 409 

[12] Amit D J, Gutfreund H and Sompolinsky H 1985 Phys. Rev. A 32 1007 

[13] Bolle D and Mallezie F 1989 J. Phys A 22 4409 

[14] van Hemmen J L, Grensing D, Huber A and Kiihn R 1986 Z. Phys. B 65 53 

[15] Coolcn A C C and Ruijgrok TH W 1988 Phys .Rev. A 38 4253 



15 




Figure 1: The overlaps for the Mattis states as a function of the temperature 
T — 1//3 for J 1 — J 2 — J3 = 1. States represented by filled symbols are only 
present for the model with linked patterns. 
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T=0.1 



T=0.7 




Figure 2: Flow diagrams for the model with unlinked patterns at several temper- 
atures. Filled symbols denote different states: a circle for Omm, a diamond for 
Imm, a square for mmm and a triangle for the symmetric state smm defined in 
the text. 
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T=0.1 



T=0.4 




Figure 3: Flow diagrams for the model with linked patterns at different temper- 
atures. Filled symbols stand for different states: a circle for mOO, a square for 
mmm, a diamond for 000 and a triangle for the asymmetric state amm defined 
in the text. 




Figure 4: Flow diagrams for the model with linked patterns at different temper- 
atures. Filled symbols stand for different states: a square for mmm, a diamond 
for 000 and a crossed circle for mmO. 
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The phase diagram of the AT neural network 



mmO (crossed state) 




Figure 5: The (5J\ — f3J^ phase diagram of the ATNN model with linked patterns. 
The dark grey area represents the 000 phase, the light grey area the full retrieval 
phase mml and the white area the partial retrieval phase 00/. Thick full lines 
indicate continuous transitions, thick dashed lines discontinuous ones. 




Figure 6: The overlap as a function of a rescaled T for the state mmm in the 
ATNN model with linked patterns (filled diamond) versus the overlap for the 
Mattis state in the Hopfield model (empty diamond). Respective temperatures 
are denoted by Tat and T#. 



